Checking for missing files in parallel #224
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for silly-keller-664934 ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
Codecov Report
@@ Coverage Diff @@
## master #224 +/- ##
========================================
- Coverage 87.0% 87.0% -0.1%
========================================
Files 29 29
Lines 1930 1937 +7
========================================
+ Hits 1681 1686 +5
- Misses 249 251 +2
|
ejm714
reviewed
Sep 7, 2022
There was a problem hiding this comment.
Code looks great and 5-6 minutes instead of 2 hours is a fantastic improvement!
I agree it's worth testing with local files as well. Can you do that? You can just mock it by downloading one sample videos, making 100 copies or so (just a little bash script would do), and then creating a simple labels file, e.g.
filepath,label
vid1.mp4,gorilla
vid2.mp4,gorilla
vid3.mp4,gorilla
vid4.mp4,gorilla
AllenDowney
changed the title
|
Confirmed that it works with 22 local files (a convenience sample of videos) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #216
Checking for missing files is slow with goofys. It seems to make lots of small queries to the file system. Running them in parallel with
pqdmis much faster. The speed depends on the state of the file system cache, but we can check 246,000 files in 5-8 minutes, compared to about two hours the slow way.Using
pqdmwith threads is faster than with processes. Using 16 threads seems to be fast and robust. With more threads, things go faster, but you start to see unpredictable I/O errors.If an error occurs, it falls back to the slow way.
This fix has only been tested with video files that are mounted from S3 using goofys. It might be good to test with videos stored in a local file system, too.